COMPACTNESS OF THE 9-NEUMANN OPERATOR AND COMMUTATORS OF THE 
BERGMAN PROJECTION WITH CONTINUOUS FUNCTIONS 



MEHMET gELiK AND SONMEZ SjAHUTOGLU 

ABSTRACT. Let H be a bounded pseudoconvex domain in C,n > 2,0 < p < n, and 1 < q < 
n — 1. We show that compactness of the 3-Neumann operator, Np /Cj+ i, on square integrable (p, q + 
1) -forms is equivalent to compactness of the commutators [Pp t q,Zj] on square integrable 3-closed 
(p, <j l )-forms for 1 < j < n where Pp /1? is the Bergman projection on (p, (j)-forms. We also show 
that compactness of the commutator of the Bergman projection with functions continuous on O 
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percolates up in the 3-complex on 3-closed forms and square integrable holomorphic forms. 



The purpose of this paper is to characterize compactness of the 9-Neumann operator on 
^ ; square integrable (p,^)-forms. Theorem [T] provides six equivalent statements, for q at least 2, 
on bounded pseudoconvex domains. However, the important special case of functions, namely 
(0, 0)-forms, remains open. In Remark [3] we discuss why our proof breaks down in this case. 

Compactness results in the 9-Neumann problem have a long history; we refer to a recent book 
by Straube [StrlOJ for a detailed discussion. We note here that most known results provide con- 
ditions for compactness in terms of the boundary geometry. It is also useful to characterize 
compactness in functional analytic terms. For example, Catlin and D'Angelo |CD97[ Theorem 1] 



used compactness of the commutators [P, (p] of the Bergman projection P and certain multiplica- 
tion operators (p in conjunction with a complex variables analogue of Hilbert's 17th problem (see 
also flD'Alll V In the same paper they showed that compactness of No,i implies that the commu- 
tators [P, M] are compact for all tangential pseudodifferential operators M of order 0. D'Angelo 
then asked the following question: 

Question 1. Can one characterize compactness of the d-Neumann operator in terms of commutators 

This question is appealing because of its connection to operator theory as well. The Hankel 
operator, Hq, with a bounded symbol <p is defined as H<p{f) = (I — P){<pf) where / is a square 
integrable holomorphic function. Using Kohn's formula, P = I — d No,i9, one can compute that 



H(pf = 9 NQ i \d{f(p). Using this formula Cuckovic and §ahutoglu |C§09| studied how bound- 
ary geometry interacts with Hankel operators. They showed that, on smooth bounded convex 
domains in C 2 , compactness of can be characterized by the behavior of cp on analytic discs 
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in the boundary. Here (p is smooth up to the boundary. It would be interesting to know if this 
characterization still holds in higher dimensions. 

On convex domains the relation between the compactness of the commutators and of the 9- 
Neumann operator has been fairly well understood: if O is a bounded convex domain, then 
compactness of Np„+i is equivalent to compactness of the commutators [Pp,q,<p] on the space of 
(p,cf) -forms with square integrable holomorphic coefficients, for all functions (p continuous on 
O (see IStrTOl Remark (ii) in Section 4.1]). 

In our previous paper |<J§12| we answered Question [T] on non-pseudoconvex domains. To be 



more precise, we constructed a smooth bounded non-pseudoconvex domain in C" (for a given 
n > 3) for which the commutators [P, (p] are compact (on square integrable functions) for all (p 
continuous on the closure of the domain yet the 9-Neumann operator N04 is not compact. In 
this paper we will consider the important pseudoconvex case and establish a decisive result on 
forms in Theorem [U 

We would like to thank Zeljko Cuckovic, John D'Angelo, and Emil Straube for valuable com- 
ments on a preliminary version of this manuscript. 



1. Background and Main Results 

Let O be a bounded pseudoconvex domain in C M .We denote the square integrable (p, ^)-forms 
and the 9-closed (p, ^)-forms on O by L^^(O) and K^^(Cl), respectively. The 9-Neumann 
operator, N P/i? , is defined as the solution operator for ^p,qU = v where D p/Cj = 99* + 9*9 on 



L, \(Oj and 9 is the Hilbert space adjoint of 9. Hormander |H6r65| showed that N Pi q is a 



bounded operator when O is a bounded pseudoconvex domain. Kohn in [Koh63j connected the 



Bergman projection Pp^ on (p, ^)-forms with the 9-Neumann operator by the formula 

Pp,q = 1 - 9 N M+ i9. 

We note that P P/Cj is the orthogonal projection onto K^^(Ci), the operator Pq,o is the classical 
Bergman projection P, and K^ QQ ^(Ci) (also denoted as A 2 (Q)) is called the Bergman space. We 
refer the reader to HCSOlllStrlOfl for more information about the 9-Neumann problem and related 
issues. 

The commutators of the Bergman projection with multiplication operators can also be written 
in terms of the 9-Neumann operator. One can use Kohn's formula to compute that 



[P P ,^<P]f = -d*Np iq+1 d(pAf 

for / £ jK? JCl) and (p € C l {D). Furthermore, compactness of Np iCj+ \ on L? +1 x(Q) implies 
that [Pp, q ,<p\ is compact on L? JCi) for all <p £ C(O) (see IIStrlQl Propositions 4.1 and 4.2]). In 
the following theorem we show that the the converse is true when q > 1. 
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Theorem 1. Let Clbe a bounded pseudoconvex domain in C n ,n > 2,0 < p < n, and 1 < q < n — 1. 
Then the following are equivalent: 

i. N M+ i z's compact on L? ^(Q), 

ii. 9*N M+ i z's compact on K? ^(O), 

iii. [Pp r q,Zj] is compact on K 2 ^(0)/or all 1 < j <n, 

iv. [Pp /Cj ,Zj] is compact on L? ^(Ti) for all 1 < j < n, 

v. [Pp /( j, </>] z's compact on L 2 x (O) /or a// ^ G C(O), 
vi. [Pp,^, </>] z's compact on K 2 ^ (O) /or all (p G C(O). 

We would like to mention that Haslinger in [Has08, Theorem 3] proved equivalence of iii., iv, 



v., and vi. in Theorem [T]in case of p = q = 0. 

One consequence of Theorem [T] is the following observation: To conclude that N P/ q is compact 

— * — * 

it is enough to verify compactness of 9 N Pi q only, instead of verifying compactness of both 9 N Pr q 
and d*N P/ q + i (see UStrlOl Proposition 4.2]). 

Remark 1. We note that compactness of N Pi o on the orthogonal complement of A 2 (Ci) is equiva- 
lent to compactness of N Pi \ on L^^(Q). This can be seen as follows: N Pi o = (d*N P/ i^J (d*N Pi i^J . 

This formula shows that compactness of N Pi o implies compactness of d*N Pr i. Then Lemma |3] im- 
plies that d*N Pr 2 compact. Finally, Range's Formula (see flStrlO , p. 77] and |Ran84] ) for q = 1 



implies that N P/ i is compact. On the other hand, compactness N Pi \ implies compactness of 
d*N Pi \. In turn, the formula N P/ o = (d*N P/ i^j (d*N P/ i^ shows that, in this case, N P/ o is com- 
pact on the orthogonal complement of A 2 (Q). 

Compactness of the 9-Neumann operator percolates up in the 9-complex (see Remark[6]). The 
following theorem shows that the same is true for the commutator of the Bergman projection 
with a function continuous on the closure of the domain. 

Theorem 2. Let Vibe a bounded pseudoconvex domain in C n for n > 2,0 < p < n, 1 < q < n — 1, and 
<p € C(O). Assume that [P Pi q, <p] is compact on K 2 v (O). Then [P M +i, <p] is compact on K 2 +1 v (O). 

Remark 2. Compactness of [P p ,q, <p] on K 2 s (O) does not necessarily imply compactness of [P p ,q, (p] 
on L 2 p^(Ci), in general. One can construct a counterexample as follows: Let Q C C n be the 
polydisk, < p < n,0 < q < n — 2, and consider (p = Zj for 1 < / < n. Then [P p ,q,zj] = on 
K 2 p^(n), hence compact. On the other hand, if [P P/ q,Zj] were compact on L^^(n) the adjoint 
[P p ,q, Zj] * = — [P Pr q, Zj] would be compact on L 2 ^ ^ (O) as well. We note that 

-[ p p,q>Z]\g = ~d*N pq+1 (dzjAg) 
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for g G s(O). In addition, one can show that any / G A? +1 -j(Q) can be written as / = 
E"=i /;' A dzy where /y G A? JO) and £* =1 ||/y|| 2 = ||/|| 2 (see the proof of Corollary |2]in the next 



section). Then we have 



9*N M+ i/=(- 1 ) P+ ' ?+1 E[ P M^ j ]/ r 

7=1 



Therefore, if [P P/(? ,Zy] were compact on L? s(O) for 1 < j < n the operator 9*N M+ i would 
be compact on A 2 ^ ^(fl). However, this contradicts 1FS981 Theorem 1.1]. We note that even 
though HFS981 Theorem 1.1] is stated on L 2 ^(Q) the proof only uses forms with holomorphic 
coefficients and the proof is valid on (p, ^)-forms as well (see remarks 2 and 4 in [FS98, pg. 638]). 



2. Proofs of Theorem [Hand 

The proof of Theorem [T] will be based on several lemmas whose proofs are elementary if one 
is familiar with the basics of the 9-Neumann problem. 

Lemma 1. Let Ci be a bounded pseudoconvex domain in C 1 for n > 2 and g £ K 1 ^ +1 ^(n) where 
< p <n and 1 < q < n — 1. Then there exist gj G K 2 p x (O) /or 1 < j <n such that 

n n 

S= Y^gjAdzjand £ ||gy|| < 
7=1 7=1 

Proof. LetO < p < n, 1 <q < n — 1, and / = 2^ |j| = „ \j\ =Cj fijdzi Adzj = 9 N P/Cj+ ig. Then we can 
write 

« 

/= Efi Ad Zj 

7=1 

where //s are square integrable (p, ^ — l)-forms so that there are no common terms between 
fj A dzj and A dz^ if j ^ k. This can be done as follows: Let V denote the adjoint of the exterior 
multiplication. That is, if / is a (p,^)-form dzj V / is a (p,q — l)-form such that (h A d.Zj,f) = 
(h, dzj V /) for all h G CT - (C M ). Then we define 

fx =dz 1 Wf 

fj = dzj V f/ - £j/ fc A rfz fc j for / = 2,3, . . . ,n. 

Namely, f\ is defined by collecting all terms that contain dz\ and writing their sum as f\ A dz\ . 
Then we define /? by collecting the terms in / — f\ A izi with rfz2 and writing their sum as 
fi A rfz"2 etc. Since dg = and / is in the range of 9*, we have 9/ = g and 9*/ = 0. So / is in 
the domains of 9 and 9*. Also since /; consists of terms fij for some \I\ = p and |/| = q, "bar" 
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derivatives of f/s come from "bar" derivatives of/. Then 



E 



dzi 



< 



E E 



\I\=pM\=lk=l 

This fact together with [StrlOl Corollary 2.13] imply that 

' < P/ll + P /I 



dzi 



E 

Hence, ||3/y|| < \\g\\ for every; and 



dzi 



g = 9 / = E d fi A dz i- 
Therefore, if we define gj = dfj we have g = Ylj=\ gj A ^z/ and E" =1 



< 



Iff I 



□ 



Remark 3. The main idea for the proof of Theorem [T] depends on Lemma [T] and failure of Lemma 
[T] for q = is the reason why Theorem [T] is still open for q = 0. One can see that Lemma [T] is 
not true for q = as follows: Let g = (p{z\)dz\ where (p is a non-holomorphic function that is 
smooth on O. Then g is 9-closed but there is no holomorphic function g\ such that g = g\dz\. 
Also it is interesting that the proof of Lemma [T] requires the existence of (p, q — 1) -forms (that is, 
two form-levels below the starting form-level). 



The following lemma shows that the converse of [StrlO, Proposition 4.1] is true. 



Lemma 2. Let Cibea bounded pseudoconvex domain in C n , n > 2, and 0<p<n,l<q<n — l. Then 
[Pp,q, Zj] is compact on ^ (O) for all I < j < n if and only ifd*Np iCj+ \ is compact on L? (O). 

Proof. We only need to prove one direction as compactness of 9 N M+ i on L? +1 n(Q) implies 
that [P Pr q,Zj] is compact on 9-closed (p, ^)-forms for 1 < ; < n (see [StrlO, Proposition 4.1]). 



To prove the other direction, assume that {g k } is a bounded sequence in +1 (Cl). Then 
Lemma [l] implies that for each k there exist 3-closed (p,q) -forms g^ for 1 < j < n such that 



< 1 


g k \\ 


llOj II r-j 1 





. Then we have 



= (-i)^ +i u F ™>m))- 

Furthermore, if [Pp,q, Zj) is compact on 9-closed (p, g)-forms for 1 < j < n, sequences { [Pp,q, Zj] (gy ) } 
have convergent subsequences for each /. Hence 9*N P/(?+ i is compact on +1 %(fi). On the 

other hand, compactness of d*N Pi q + i on 9-closed forms is equivalent to compactness of 9*N M+ i 
on ^ (O) as 9*N P/(? +i vanishes on the orthogonal complement of ^ (O). □ 
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Remark 4. It is interesting to observe that if q — n — 1 then the commutators [P„ /M _i,Zy] are com- 
pact on K 2 w (O) on any bounded pseudoconvex domain O with sufficiently smooth bound- 
ary. This is a consequence of the fact that the 9-Neumann problem on (p, n) -forms is the classical 
Dirichlet problem. See flCSOli Remark after Corollary 5.1.7.]) 

Remark 5. It is still unknown if Lemma|2]is true when the commutators are restricted on A 2 % (O), 
the square integrable holomorphic (p, ^)-forms, for < p < n, and < q < n — 1. That is, it is 
not known whether compactness of [Pp,q, zj\ on A 2 ^ s (O) for all / (equivalent to compactness of 

3*Np /(7+ i on A 2 (O) by flStrlOi Remark (ii) in Section 4.1]) imply compactness of 9*N M+ i on 

Lemma 3. Let Clbe a bounded pseudoconvex domain in C n ,n > 2,0 < p < n, and 1 < q < n — 1. 
Then compactness ofd*Np r q on K 2 ^ (O) implies that d*Np /Cj+ i is compact on K 2 +1 s (O). 

Proof. Let {g k } be a bounded sequence of 9-closed (p, q + l)-forms. Then by Lemma[T]there exist 
9-closed (p, g)-forms g k 's such thatg fc = Y^=\S) ^ d-Zj an d EyLi llgy II ^ ll& fc ||- 

Let us define f k = Ylj=\ d*N Pi q{g k ) ^ dz~j- Then df k = g k and compactness of 9*N M implies that 
{f k } has a convergent subsequence. Therefore, 9 has a compact solution operator on (p,q + 1)- 
forms. Hence, the canonical solution operator, 9*N M+ i, is compact on K 2 +] x (O). □ 

Remark 6. One corollary of Lemma |3] is the well known fact that compactness of Np f q implies 
compactness of Np /Cj+ i (see [|Strl0l Proposition 4.5]). One can see this using Lemma |3] together 
with Range's formula ( HRan84IO : 

n m = (a*N M )*a*N M + a*N M+1 (a*N M+ i)*. 

Lemma 4. Lei Clbe a domain in C",0 < p,q < n and (p,tp € L°°(n). Assume that [Pp,q,(p] and 
[Pp r q, tp] are compact on L 2 ^ (O). TTzen [Pp /(? , </>i/?] is compact on L? ^ (Q). 

Proof. Assume that the operator [P M , (p] and [Pp^, t/>] are compact on L 2 ^ ^ (Q). For / € ^ (O) 
one can compute that 

[PwMf + <P\ P P,V W = ( P P^ - 4>P M W + ~ <PPp,q)f 

= Pp,qHf ~ M P P,cif 

= [PwW>]f- 

Hence, 

(i) [Pp*<N>]f = [Pwfltf + <PlPp,i>W- 

Therefore, [Pp,q, <p$] is compact on L 2 s (Q) because the terms on the right hand side of ((TJ) are 
composition of bounded operators with compact operators on L 2 s (O). □ 



COMPACTNESS OF THE 3-NEUMANN OPERATOR AND THE COMMUTATORS 



7 



Let us define 

r(/M) = {<*> e C (^) : t P M^] : L ( M )(°) L ( M )(°) is compact}. 
Corollary 1. Le£ Clbea bounded domain in C n . Then T(p, q) is a C* subalgebra (with identity) o/C(H). 

Proof. The fact that compactness is preserved under the operator norm topology implies that 
T(p,q) is a closed subspace of C(O). Lemma [4] implies that T{p,q) is a subalgebra of C(O) and 
1 € I{p,q) because [P«,a/ 1] = 0. Finally the fact that T(p,q) is closed under conjugation follows 
from the formula [Pp,q, <p] * = — [Pp,q, W\ and the fact that an operator is compact if and only if so 
is its adjoint. □ 

We note that Corollary [TJ gives a characterization for compactness of N P/ q for q > 1 in terms of 
C* algebras. Namely for 1 < q < n — 1 the operator N P/q+ i is compact if and only if T(p,q) = 
C(Q) (compare to lISaMl ). 

Proof of Theorem^ We note that i. =>- ii. is known (see [StrlO, Proposition 4.2]) and ii. => iii. is 
easy because [Pp,q,Zj](f) = -<T N p , q+1 (dzj A /) for / G K 2 pcj) (n). 

Now we will prove iii. => L: Compactness of [P P/ q,Zj] on K 2 ^^(d) for 1 < j < n, by Lemma 
is equivalent to compactness of d*N P;Cj+ \ on K? +1 n(0). On the other hand, if q = n — 1 
then compactness of 9 N PiCj+ \ is equivalent to compactness of N Piq+ \. In case 1 < q < n — 2, 
Lemma [3] implies that compactness of d*N Piq+ \ on K 2 ^ +1 ^(n) implies compactness of d*N p>q+ 2 
on (O). Then UStrlOi Proposition 4.2] implies that N Piq+ i is compact on +1 j (O). 

The implication ii. => iv. follows from the fact that d*N p iCj +i vanishes on the orthogonal com- 
plement of K 2 +1 s (Q) and [StrlOi Proposition 4.1]. 

We prove the implication iv. =>- v. as follows: By Corollary [TJ we know that T(p, q) is a closed 
subalgebra with identity (of C(O)) that is closed under conjugation. Then the assumption that 
the commutators [Pp,q,Zj] are compact on L^^(Q), for 1 < j < n, together with the Stone- 
Weierstrass Theorem imply that T(p,q) = C(O). That is [P p ,q,<p] is compact for all cp € C(O). 

Finally the implications v. =>- vi. and vi. iii. are obvious. □ 

Proof of Theorem^ If q = n — 1 then ^(Q) = L^^(O) and [P PiCj+ \, <p] is the zero operator, 
hence compact. So for the rest of the proof we may assume that n > 3 and 1 < q < n — 2. 

Let g G K 2 (O). Then Lemma [TJ implies that there exist gj G K 2 ^ (O) for 1 < / < n such 
that 

n n 

S= TjSi^dZj and £ ||#|| < 

7 =1 ;=1 

Now we will show that 

(2) [P M +l^]g = (I - P M +l) \^E([Pp, q ,<P]gj) A 
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Since both sides of (0 are orthogonal to K? +1 ^(Q) we only need to show that for any h G 
L? + ^ (Q) that is orthogonal to (O) we have 

[P M +l,*]g - (I - P M +l) ^E([P M ^]g 7 ) A dz ; j ,^ = 
where (, ., ) denotes the inner product on L? +1 s (O). One can compute that 

lP p ,c,+i,(p}g- (I - P M +i) f E([P M ^k 7 ) Arfz ; j = - fc) - (f^P P , q ((pgj)AdZj,h 

n 

The fact that 9(/ A dzj) = (9/) A rfzy implies that the (p, g + l)-forms Pp,q(<pgj) A dzy are 9-closed 
for j = l,...,n. Therefore, we have ( LyLi Ppjitygj) A dzj,hj = and the equality © is proven. 

Let C (O) be a bounded sequence. Then Lemma[T]implies that for each k and 1 < 

; < n there exists g 1 - G x(O) such that g /c = 7J" =1 gy A rfzy and 7Jy =1 ||gy|| < ||g fc ||. Furthermore, 
compactness of [Pp /(? ,0] on K| (q) implies that for each 1 < j < n the sequence {{Pp,q/(p]gj} 
has a convergent subsequence. Now using (0 we conclude that the sequence { [Pp,q+i, <p]g k } has 
a convergent subsequence. Hence [Pp^+i, <p] is compact on K? +1 s (O). □ 

Corollary 2. Lef Clbea domain in C n , < p < n,0 < q < n — 1, ant? € C(O). Assume that [Pp,q, <p] 
be compact on A? s (O). Tfoen [P M+ i, <p] is compact on A? +1 s (Q). 

Proof. First of all, we note that (0 is valid on all domains, possibly unbounded or non-pseudoconvex. 
With this in mind, one can prove Corollary |2] the same way as Theorem |2] except, instead of 
Lemma Q] we need the following fact: Let g G A? +1 ^(O),0 < p < n, and < q < n — 1. Then 
f or 1 < j < n there exists gj G A? .(O) such that g = YJj=\gj A ^zy and ||g|| 2 = 7J" =1 \\gj\\ 2 - In 
fact, one can define g\ = dz\ V g and gy = dzy V (g — L( = i &fc A dz^j for ; = 2, 3, . . . , n. □ 

Remark 7. Corollary [2] is stated for any domain in C n and < q < n — 1 while Theorem [2] 
is stated for bounded pseudoconvex domains and 1 < q < n — 1. Also Corollary |2] implies 
that on bounded pseudoconvex domains compactness of the 9-Neumann operator on square 
integrable holomorphic forms percolates up in the 9-complex. One can see this implication by 
observing that on a bounded pseudoconvex domain O compactness of the commutators \Pp,q, <p] 
on A? s(O) for all cp G C(Q) is equivalent to compactness of N M+ i on A? +1 s(0) (see, for 
example, UStrlOl Remark (ii) in pg. 75]). 
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